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ABSTRACT 



This paper deals with two question^:" (1) 
applications that might be made of microcomputers in 



what are the 
evaluat i pn? 



the^ 

^nd (2) what are the kinds of problems (personnel , organizational , 
and techniqal) that must be c^v^rcome if microcomputers are to be used 
ef feet ively'r*'' The applications'' include proposals and planning, 
management, data collect i onv data analysis and interpretation, and 
reporting. Several types of microcomputer software couldbe usfed in 
these applications: woi^d Tprocess i ng , data bases, "test genera't i ons , 
calculation or stat i st ical' an^lys i s , graphic presentation, and 
telecommunicat ions . fohe ways-in which these types of software can be 
effectively usfedin evaluation act ivi ties "are discussed in the first 
part of this report. The' second part addresses the personnel^ 
organizat ianal; and techn i cal i ssues surrounding microcomputer i^se. 
In addressing these issues in their own setting, and by following a 
systematic, problem-solving approach to thei^r own use o3e 
microcomputers, evaluation researchers can gain knqwledge ^nd i^kills 
needed to help otheVs as well. (BW) - ^ 
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PREFACE 



The Research on Evaluation Program is a Northwest Regional 
Educational Laboratory project of re^earchr development r testing r 
and training designed to create .new evaluation methodologies for 
use in education. This document is one of a series of papers and 
reports produced by program staff r visiting scholars r adjunct 
scholars, and pro'ject collaborators — all members of a cooperative 
network of colleagiies working on the development of new 
methodologies* 

What are the appliiDations that might be made of microcomputers in 
eva^uatioOf and what are the kinds of problems (personnel r 
organizational anQ technical) that must be overcome if 
microcomputers are to be used effectively? These (Questions arfe 
addressed in this brief introduction to the use of microcomputers 
in evaluation work. ^ 

1^ . Nick L. Smith r Editor 

Paper and Report Series 



» 




CC3NTENTS 



Tasks and Programs 

*. 

Planning and Proposal Development . . . 

Study Management . . ^ 

Data Collection and Data Analysis . . . 
Reporting . . . , 

People, Or g?inizational , and Technical Issues 

Peop]^e Issues ^ . 

Organizational Issues * 

Technological Issues 



/ 



/ '6 ^ 



MICROCOMPUTERS AND EVALUATION RESEARCH: 
POTENTIAL BENEFITS AND PROBLEMS 

( 

■4 

Microcomputer J - The very word causes both fear, and longing in 
the heart of the most seasoned evaluator, This is a natural and 
appropriate response since both great potential and important 
problems are associated with the use of microcomputers in 
evaluation research. 

Microcomputers are not ends in themselves, btl|; are rather the 
means for performing tasks more' efficiently and effectively. 
Microcomputer hardware components (e«g«r central processing unit, 
key board r (nonitor, disk driveSr printer , modem) embody raw 
potential « It is the progr2ans or software that direct the 



operation and integration of \these components for a particular 
purpose that allows the user to realize their potential*^ The 
potential of using microcompiiter programs for accomplishing the 
tasks related to evaluation research is the topic of the first 



section of this paper ♦ There are, however r' some potential 
problems inherent in applying microcomputer technology ta 
evaluation research. These probletas revolve around pebpler 
organizational, and technological issues. In the second part of 
this paper these issuei^ are explored. 

V • 

Tasks and Programs 

^ Evaluation research typically follows a process which begins 
with the clarification of a problem and the development of a plan 
or, more formally r a proposal for a particular way of addressing 
the problem. That plan usually contains designs for managing the 
study, forv^data collection<r and for data analysi"Sr synthesis, and 



interpretation. Some provisions for reporting the results of the 
study are usually included as well. Microcomputers can be used 
to facilitate both planning and implementing each of these steps. 

Figure 1* is a matrix showing the relationship between the 
steps in the evaluation, research process and common types of 
microcomputer software. Programs may be written by the user with 
the aid of a programming language (e.g.r BASIC, Pascal, COBDL) . 
They may be in yie form of general purpose pre--pac?kaged programs 
which are designed to accomplish a class of tasks such as is 
subsumed by the concepts of word processing or data base 
management. These packages can be tailored to meet the 
requirements of many different tasks within a group of, tasks. 
Programs may also be designed to accomplish a c^tain\kind of 
task in a particular way with little opportunity for 
modification. This is the case with sortie statistical packages 
and some accounting packages , for example. 



Figure 1 

Matrix of Evaluation Research Tasks 
and Related Type^ of Software Programs 
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This section is organized around the ac^vities that are 
listed down the side of the matrix in Figure 1. The discussion 
relates the characteristics of the different types of programs 
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and their utility to t^e steps of the evaluation research 
process. For more extensive discussions of the various types of 
microcomputer programs, see Gray (1984, in press), Gray and Deck 
(1983^), Spuck and Atkinson (1983), Tall^y (19^3) and the various 
prograuQ specific reviews in coipputer magazines referenced in 
these sources^ , 

Planning and Proposal Development 

It is in situations where repetitive activity occurs that 
microcomputer progr£uns are of the gW^test' potential benefit. 
These situations include producing multiple drafts of documents, 
making repeated calculations, and reporting similar, information 
in slightly different format?. Planning and proposal development 
usually include all of these. 

The initial conceptualization of a study typically relies on 
the use of words. A statement of the problem as the researcher 
sees it,, a review of the relevant literature, the description of 
the proposed solution are all standard partb of research plans or 
proposals. And, they usually have to be presented in a 
pre-specif ied format. These often go through many drafts in the 
pj^anning process. Programs that help one manipulate words, that 
is, word processing programs, make microcomputers useful in 
planning and proposal development. They have features that make 
the typing of text and its 'editing, formatting and'printing very 
convenient. . 

Developing a proposed budge^t is also an important part of the 
planning process. All-purpose or generic electronic spreadsheet 
programs provide a ledger format for use in laying out the line 
items of a budget. As the details of a plan are finalized, its 
budget changes, too. The ability to automatically recalculate 
budget figures based on pre-defined relationships between the 
cells in a spreadsheet means that it i^a simple matter to make 
budget projections under changing conditions with spreadsheet 
programs. 
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Large data banks and data base. programs c$n be useful for 
doing the literature review at the initial stages of a study. 
Data banks such as that represented by ERIC can be very helpful 

9 

in finding research related to a particular study. Through the 
use of telecommunications, various data banks can be searched. 
The information frc^n these sources and from more conventional 
manual searches can be entered into a ^ata base of references 
organized by author, subject, and so on. Prom these records, 
specific Information and references can be extracted for 
inclusion in a proposal, for example, as direct quotations or as 
general references. Data base programs provide the flexibility 
to take one standard set of information and organize It ip many 
different ways. 

Setting up the sch^edule of activities for a study can be 
accomplished using a general program, such as one for word 
processing, or a specific project management program. What is 
needed is some way to organize the events of the study along a 
time line. This can be done under column headings such a@ event, 
dater gost, and responsibility using a word processing program. 
It can also be done in the form of a PERT or pther such chart. 
Using the microcomputer to set up and revise ti^e study schedule 
provides an advantage, in that as conditio^ change, new versions 
of the schedule can b^ generated and ptinted. This alleviates 
the necessity of having to manually reproduce those parts of the 
schedule that stay the same from one draft to the next. 

9 

Study Management ^ ^ 

The ongoing' management of a study encompasses many different 
tas^s. Thesa include developing^ and monitoring data collection 
techniques, accounting for program expenditures, maintaining 
study timelines, and reporting prograun prog|re3s. 

The develofMnent of data collection procedures can in many 
ca^s be aided by microcomputers. In Data for Decisions , 
Hoaglin, Light, McPeek, Mostellei^, and Soto (1982) list three 
types of studies: cause and effect, status quo, and prediction. 
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Each type has several designs related to it» For example, cause 
and effect studies include experiments as well as comparative 
observational studies • Status quo studies include sample 
surveys^ longitudinal studiifeSr and case studies. Prediction 

studies include simulations/ mathematical modeling, and studies 

* * 

of introspection^ and advice « ^ 

The test and observation protocols used in experimental 
studies and the survey instruments used in stati/s quo Studies can ^ 
be generated r formatted r and printed using generic word 
processirjig programs • In some cases a data base management 
progreun may be used to maintain, sort, and select among 4 large ^ 
pool of potential items for a particular purpose. Special canned 
instrument development programs which are a form of data base 
programs are also available r or one can be treated for a specific ^ 
bank of items • 

One of the most valuable ^uses of microcomputers m ^ 
educational settings is to engage learners in simulations. Such 
programs can be Used in prediction studies, <for example, to 
evaluate likely future administrative behavior throught^ 
"in-basket" simulations. As the power and speed of micro- 
computers continue to grow, they become even more attractive 
tools for use in the development of mathematical modeling 
procedures, another form of prediction study . Increased power 
and speed also will make them useful in regard to complex 
procedures^ which involve the repeated manipulation of information 
such as introspection and advice studies. ^ 

-^Clearly, word processing data base management r and various 
special programs can make mioi^t^oroputers valuable tools in the 
refinement of data collection procedures. Similarly, electronic 
spreadsheet programs and accounting programs aure convenient and 
effective tool3 for monitoring program costs and projepting 
expenditures. The progreuns that were used to set up the study 
budget and th^ study timeline' in the first^place can facilitate 
updating costs and target dates. These provide efficient y 
vehicles for monitoring data collection and for managing the 7 
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study as a whole* Of course, word processing programs are useful 
in the ongoing reporting of study progress. . 

Data Collectiort" and Data Analysis 

In addition to being useful in the development, of data collection 
procedures, some softii^are can also facilitate the collection and 
analysis of dat£i. There are programs which can be used by 
interviewers to conduct telephone surveys* And there are 
programs that collect data directly from subjects in a more or 
less obvious fashion depending on their purpose. Many of these 
programs have routines built into them to analyze the data 
(Collected • ' . 

A growing body o^ statistical software for data analysis ia 
accuAiulating.>L These range from simple inexpensive programs to 
those which emulate the complexity of mainframe programs. Almost 
every type of statistical analysis can be found, including 
exploratbry ^jiata analysis techniques (see, especially. Carpenter, 
J., Deloria, D. , and Morganstein, D. , 1984). 

Many statistical programs can provide numerical syntheses of 
data, and also graphic presentations. In addition, tKe^^^e 
graphics progreuns which can take a data file and create line, 
bar, pie, and three- dimensional graphs. Some spreadsheet 
programs are integrated with very sophisticated graphics 
programsS^ Using graphics can be a great assistance in the 
interpretation of a study's results by the researcher, as well as 
others. 

For data that include information other than numbers, data 
base or file management programs can be very useful in 
collection, analysis, and interpretation. Such programs 
facilitate the sorting ^nd selection of data with particular 
characteristics, for example, subjects from certain areas or with 
particular interests. If, after analysis, other^character istics 
appear to be important, a^ data base^ program can be used to 
reorganize the records in a file artd present the results from a 
different perspective. 



* If data have been collected at remote sites, it is often 
advantageous to be able to telecoramunicate/ them to a central 
location* They can then be acted upon by the appropriate* 
progrsuny This may mean a mainframe program if the data set 



/ ' and/or analysis is too complex for a microcomputer. In such a 

c^se, a microcomputejc can act as a dumb terminal to just transmit 
and receive information/ or as^a smart terminal in order to 
interact with the larger, more powerful computer. 

- ^ ; Reporting 

Reporting, like planning/ is an activity which relies mai^nly 
on words. However, in addition to word processing] progr£uns, ^ 
graphics programs can greatly enhance the effectiveness of a 

written or oral report. There are some integrate/) software 

i 

progrcuns that combine word processing, data bas^ management or 
spreadsheets, and graphics programs. Such progrsuos can provide 
an efficient way to pull together the results of a study. But 
even the separate output from a^ word processing program, and a ^ 
' data base, spreadsheet, or graphics progreuQ can be easily 

combined to produce effective reports. Reports thus generated 
can be transmitted via telecommunication networks to sponsoring 
agencies or other interested audiences. 

In summary, custom designed, generic, or special purpose 
programs make it possible to use microccMnputeriS at every step of 
the research process. However, the availabiliq/ of such programs 
does not mean that the adoption and implementation of 
microcomputer use will be easy. In the next section, the 
problems surrounding the use of microcomputers for evaluation 
research are discussed. 



People, Organizational, and 
^ Technical Issues 



The versatility and relatively low cost of inicrocompu%ers has 
given rise to a new phenomena, the personal computer^ 
Individuals cajn, on their own, purchase a very powerful computer 

. 7 
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for personal/professional use. In maay organizations r the price 
of a microcompater is within the discretionary budget of many 
depar tments « In such a situation, it is hard enough to keep an 
inventory of hardware and software r and almost impossible ^o 
controy^thefr purchase, maintenancer and use. Clearly , the 
radical technology represented by microcomputers creates some 
important people, organizational, and technological issues. This 
is no different for evaluation researchers than for any other 
professionals. 

Technology with a capital "T" is 9^1^ evident in regard to 
microcomputers that concern over "getting machines often 
overshadows "legitimate people issues and organizational issues* 
In addressing these issues in their own setting, evaluation 
researchers can gain knowledge and skills needed to help other's 



as well. 



People Issues 

Th^ people issues surrounding microcomputer use incl.ude both 
personal concerns and informal sogial group concerns. 
Individuals seem to be either fascinated by microcomputers or 
hostile to them. This may be a case of the "techno-twit" versus 
the "techno-phobe. " That is, as with any area, there are those 
who find microcomputers natural! and exciting tools, and others 
who approach them with fear and trepidation. ^ 

Many microcomputers seem designed for the " techno- twi t s. " 

There is an emphasis on progrcunming, on cryptic commands, and on 

the "gee-whiz" factor associated with the technological bell? and 

whistles of hardware and software. As a result, a new role has 
* ■ ■. > 

emerged, that of the compqter-buf f — the one person who early on 

established hlmsel/ or herself as l:he local expert.. 

The emergence of a computer-buff .^can cause tension in a 

group, because such a role is often viewed as attractive and 

because in some cases tiie person can dominate the use of the 

computer (s). However, it can also be ^ benefit tK> a group if the 

person is willing to share information and to act as a leader in 

building the group's computer- related capabilities. 



What to compu^ter izfe ? One way to think alk>ut capacity- 

building is to begin with the tasks that individuals do every 

i 

day, A three-step screening process can helf) to identify which 
ones are amenable to computer ization^ and who might be involved. 
The first step is to eljiroinater for the' time being, those tasks . 
that Are being performed in a relatively^^&f f i,cient and .effective 
manner. (If it isn't broken, don't fix it.) 

This should leave those tasks about which people have some 
dissatisfaction. From this group, eliminate, those where" the time 
and expense of computerization is not ^ feasible at present. Other 
solutions to these problems should be exj^lored before that of 



3 betore tinat or 
vdo it, conyractii 



using microcomp«rters- (e.g hiring someoi^^ to >do it, contracting 
it out, changing the task), ^afi good exam{>le here is an out-of- 
control filing system. A filing system will not automatically 
become efficient by putting it oil a computer. In* fact, you may 
end up with a sloppy computerized filing system, that is worse 
than before. Perhaps the appropriate solution is to have someone 
to get the filing system in order first and then think about the 
use of computers. 

■ ! ) 

After eliminating those tasks that do not need fixing and 
those that might better be fixed soiAe other wa^T, the final group 
.will include tasks that are likely Jbo benefit from^ 

O I 

computer izatsion. These are tasks about which there is a felt 
need for improvement and yhich are doable within a short period 
^^-2 years) . These tasks may well cluister around particular 
roles in an evaluation research office such as office/clerical ^ 
(repeated typing of report drafts), statistician (flexible and 
varied manipulation of data), consultant (development of ^ 
transparencies and handouts summarizing results) , research 
specialist (instrument design and data collection), research 
associate (proposal and report development) , and administrator 
(budget tracking and project raanageinentp. 

The planning process ^ Th0 literature on change suggests 

^ - ■ 

three steps leading to the implementation of an innovation: 

1. Clarification 

2. Adoption of Alternatives 

..3 k Planning foif Implementation 
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By going through the screening process just dfescr ibed^ >th^ 
various people in a group can become clearer about the' meaning of 
computer use for their own tasks.. They also cpme to see th^y 
alternatives to cbmputer use and the alternative ways computers 
can ba used. 

Involvement in planning for the utilization of microcomputers 
for tasks that are considered iraportantr and in need of 
improvement r will continue to' enrich people's understanding of 
computer use. In effect^ this will facilitate the de\el6pment of 
many local experts regarding the various tasks fpjc which the ' 



implementation of raicrocoraputers is warranted^' vf^^^t^^ 

^Involving people in eaph'^tage of the above ""'pj^^jcess will help 

them find personal meaning in such a change as. computer use. It 

can also help to build good interpersonal relationships as people 

see each other becoming expert in a particular computer use. 



Organizational Issues 

The process of planning for microcomputer use can be carried 
out in an informal way^ but in most .evaluation research settings 
it will be part of an organizational effort to manage the 
introduction and widespread use of microcomputers. Two related 
issues typically emerge Vhen microcomputer use is considered in 
dn organizational context. One is centralized versus 
decentralized control ^ and the second is equitable access to and 
use of microcomputers. * , . 

With large mainframe systems^ and with most mini-computer 
s5^$t;0ittsy control is centralized r for example, in the data 
processing departraerltr ^nd both access and uses aire strictly 
limited^ Microcomputers bring some of the power of these larger 
computers to individuals. In addition^ the great variety of 
generic and specialized programs', a? well as the potential for 
individuals to write their own programs^for microcomputers, meanp 
that pressure for multiple uses by many different individuals is 
likely. 
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Managing this p^ssure is not an easy task. Setting 
priorities for access and use^ balancing cV)ntrQl and flexibility, 
and maintaining use schedules ate often complicated by the fact 
that, at least at first, only a few machines are available, and , 
their current users may feel* very possessive of them. Both, 
short-range and long-range plans are needed^ to work out these 
compXlcations. . ^ 

Short-range plans (1-2 years) should focus on clarification, 
adoption of alternatives, and the development of long-range 
implementation pla|[is. Short-range plan^should include 
provisions '^or training and other experiences designed to spread 
expertise throughout the organization. 

Training can* b^ linked to the purchase of 9 few new systems 
to broaden access beyond the initial users. Theses systems should 
be designed^ to facilitate the accomplishment of the highest 
priority tasks identified by the. screening process described 
earlier. Having potential users involved in both ^shor^-term and 
long-terra planning for implementation will help to spread the 
understanding of meaningful computer use throughout an 
organization. 

Long-range plans (3-5 years) should address the following 
areas: 

1. management and coordination r^sponsibilitles 

2. timelines for gradual expansion of users and uses 

3. guidelines for systematic, ongoing staff development 

4. rules about compatabillty among systems within the 
organization, and between internal and external 
computer systems 

5. procedures for the incorporation o& new technology 

} 

In other words, organizational issues concern the integration of 
microcomputer use Into the policies and procedural structure of 
the organization in both short-term and long-term contexts (see 
Gray, 1984 May). In developing long-range plans, it is critical 
to take into account the special technological Issues related to 
widespread microcomputer use. 
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A Change in what people do and how they do it is the essence 
of technological change*, x. Microcomputer use is bound to have a 
great impact on both of these manifestations of change. A whole 
new way of thinking about things is embodied in the use of 
"microcomputers for common r everyday tasks such as writing a ^ 

r 

letter, filing information , or creating a gra^h. New concepts 
have to be learned (e.g. , documejit, merge, record) and, although 
they often are related to known concepts, they can throw anyone 
off balance when they are presented in the context of a brief 
menu of options or A set of cryptic key commands. 

The two sides of the technological coin are represented by 
what has been called "qomputer liter acy** for users and ••user , 
friendly" in regard to software and hardware. Computer literacy 
defined ^here as competent computer use , will develop as people 
are involved in (1) clarifying the meaning of computer use ir\ 
terms of the tasks they perform, (2) shaping ^adoption 



alternatives, (3) making plans for implementing the use of 
microcomputers, and (4) being Involved in the implementation of 
computer use for tasks which are meaningful to them. This is an 
ongoing staff development process that should ^begin long before 
microcomputers are introduced en masse into an organization. 

y The ••f riendliness'' ease-of-use of software/hardware 
systems can best be determined through a systematic evaluation 
process. Such a process would consider two elements. • The fil 
is the characteristics of software packages, including (a) 
written and on-screen documentation, (b) input options and 
limitations, (c) operational speed and flexibility, (d) oui 
options and limit;itions, ^ind (e) general traits such as 
compatibility with other programs, reliability, and support by 
'the manufacturer. These should be related to the second element 
the physical attributes of the hardware, such as (a) number of 
keys, their layout of the keyboard and their function (s) , 
(b) monitor screen color and resolution, (c) disk drive capacity 
(d) internal memory and speed, (e) printer width and the quality 
of the printed image. 18 
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A thotough evaluation of software and hardware should be pact 
of the process of planning for implementation (the author is 
currently developing an Evafluator'3 Guide for Microcomputer-Based 
Administrative Packages) • In this process, tasks to be perfjprmed 
„ are matched to software with relevant characteristics after which 
appropriate hardware is specified. The coordination of purchase^ 
based on this evaluation can help to insure compatibility and 
reduce unnecessary duplication. „ 

There are several other technological issues. FoA example 
the provision of service, maintenance, and support of hardware 
and software is an issue in the sense that special expertise jLs 
needed in each area. Facilities and equipment may need to be 
specially designed for microcomputer use. A major facilities 
issue is the space allocated for microcomputers. Security versus 
availability is of concern here. Because microcomputers are 
essentially stand-alone 'units, they are vulnerable to abuse, 
vandalism, and theft. However, overly tight security such as 
stickinCi them in locked, windowless^ rooms may discourage their 
use. 

Facilities issues als<^ include electx^ical requirements, that 
is, the power^ to run one or more systems with central processing 
units and peripherals, and precautions against power surges and 
power losses. Furniture is needed which maked the use of 
machines convenient and which provides an efficient arrangement 
^ of hardware. Noise reduction in areas where many keyboards or 
printers are being used must be considered. 

The security of information stored; on a microcomputer (e.g., 
flexible diskette^ or hard disk) is also a technologi6al 
concern. Access to personnel records^ student records, and 
district financial records must be protected. 



In Gray (1983) ^ thorough description is presented of thes 
and other technological factors related to organizing for 
microcomputer use in school districts. These factors are 
relevant to evaluation research uses as well and must be 
considered in making long-grange plans. 
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^ In aununaryr the use of microcomputers for evaluation research 
has many exciting potential benefits. However r some complex 
issues rewarding-people organizational^ and technological 
concerns have to be addressed to insure relatively smooth i 
implementation « Evaluation reseanrch units can ^et a' gq^jP^example 
and ban learn how to help others plan for the use of 
microcomputers by following a systematic, problem-;s< 
approach to microcomputer use* In this way the poteatial 
benefits can be maximized, and the disruptive nature of potential 
problems can be kept to a minimum. ^ 
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